Add DatabricksTaskOperator #40013
Merged
Add DatabricksTaskOperator #40013
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pankajkoti
force-pushed
the
add-databricks-task-operator
branch
from
2fb3bb0 to
058d082
Compare
|
cc: @tatiana for helping review the PR. |
Lee-W
reviewed
Jun 3, 2024
pankajastro
reviewed
Jun 3, 2024
pankajkoti
force-pushed
the
add-databricks-task-operator
branch
from
e26fa40 to
c5c3d34
Compare
Lee-W
approved these changes
Jun 4, 2024
tatiana
approved these changes
Jun 4, 2024
There was a problem hiding this comment.
It looks great, @pankajkoti , thanks for all the refactoring and improvements!
I added a minor comment on the documentation.
fdemiane
pushed a commit
to fdemiane/airflow
that referenced
this pull request
Jun 6, 2024
This pull request introduces the [DatabricksTaskOperator](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/common.py#L26) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. Unlike the `DatabricksNotebookOperator` which only allows to run Notebook tasks, the `DatabricksTaskOperator` allows running all kinds of tasks across a wide range of types including notebooks, JAR files, Python scripts, Databricks SQL queries and dashboards, Delta Live Tables pipelines, dbt tasks and more that are supported by the [Jobs API in its tasks attribute](https://docs.databricks.com/api/workspace/jobs/create#tasks). It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider. This PR also abstracts the common implementation between the `DatabricksNotebookOperator` and `DatabricksTaskOperator` into a base abstract class `DatabricksTaskBaseOperator` so that we do not introduce duplicate code for the common set of implementation methods.
syedahsn
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 7, 2024
This pull request introduces the [DatabricksTaskOperator](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/common.py#L26) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. Unlike the `DatabricksNotebookOperator` which only allows to run Notebook tasks, the `DatabricksTaskOperator` allows running all kinds of tasks across a wide range of types including notebooks, JAR files, Python scripts, Databricks SQL queries and dashboards, Delta Live Tables pipelines, dbt tasks and more that are supported by the [Jobs API in its tasks attribute](https://docs.databricks.com/api/workspace/jobs/create#tasks). It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider. This PR also abstracts the common implementation between the `DatabricksNotebookOperator` and `DatabricksTaskOperator` into a base abstract class `DatabricksTaskBaseOperator` so that we do not introduce duplicate code for the common set of implementation methods.
96 tasks
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
This pull request introduces the [DatabricksTaskOperator](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/common.py#L26) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. Unlike the `DatabricksNotebookOperator` which only allows to run Notebook tasks, the `DatabricksTaskOperator` allows running all kinds of tasks across a wide range of types including notebooks, JAR files, Python scripts, Databricks SQL queries and dashboards, Delta Live Tables pipelines, dbt tasks and more that are supported by the [Jobs API in its tasks attribute](https://docs.databricks.com/api/workspace/jobs/create#tasks). It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider. This PR also abstracts the common implementation between the `DatabricksNotebookOperator` and `DatabricksTaskOperator` into a base abstract class `DatabricksTaskBaseOperator` so that we do not introduce duplicate code for the common set of implementation methods.
pankajkoti
added a commit
to astronomer/astro-provider-databricks
that referenced
this pull request
Aug 8, 2024
As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider: 1. apache/airflow#39178 2. apache/airflow#39771 3. apache/airflow#40013 4. apache/airflow#40724 5. apache/airflow#39295 All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install `apache-airflow-providers-databricks>=6.8.0`, which includes all the contributions mentioned above. closes: astronomer/issues-airflow#715
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces the DatabricksTaskOperator
to the Airflow Databricks provider from the astro-provider-databricks
repository. Unlike the
DatabricksNotebookOperatorwhich onlyallows to run Notebook tasks, the
DatabricksTaskOperatorallowsrunning all kinds of tasks across a wide range of types including
notebooks, JAR files, Python scripts, Databricks SQL queries
and dashboards, Delta Live Tables pipelines, dbt tasks and more
that are supported by the Jobs API in its tasks attribute.
It marks another pull request aimed at contributing operators
and features from that repository into the Airflow Databricks
provider. This PR also abstracts the common implementation
between the
DatabricksNotebookOperatorandDatabricksTaskOperatorinto a base abstract classDatabricksTaskBaseOperatorso that we do not introduceduplicate code for the common set of implementation methods.
Example successful run of a DAG using the
DatabricksTaskOperatorwith the implementation from this PR^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.